Skip to content

add readFromDeltaLake transform#4027

Merged
derrickaw merged 4 commits into
GoogleCloudPlatform:mainfrom
derrickaw:20260714_deltaLakeAdd
Jul 17, 2026
Merged

add readFromDeltaLake transform#4027
derrickaw merged 4 commits into
GoogleCloudPlatform:mainfrom
derrickaw:20260714_deltaLakeAdd

Conversation

@derrickaw

@derrickaw derrickaw commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  1. Add a readFromDeltaLake transform

@derrickaw derrickaw added the addition New feature or request label Jul 14, 2026
@codecov

codecov Bot commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.09%. Comparing base (0caed64) to head (58404c0).
⚠️ Report is 5 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (0caed64) and HEAD (58404c0). Click for more details.

HEAD has 6 uploads less than BASE
Flag BASE (0caed64) HEAD (58404c0)
7 1
Additional details and impacted files 
@@              Coverage Diff              @@
##               main    #4027       +/-   ##
=============================================
- Coverage     55.68%   38.09%   -17.59%     
+ Complexity     7236      511     -6725     
=============================================
  Files          1124      195      -929     
  Lines         68447    11759    -56688     
  Branches       7721     1176     -6545     
=============================================
- Hits          38117     4480    -33637     
+ Misses        27856     6914    -20942     
+ Partials       2474      365     -2109
Components Coverage Δ
spanner-templates ∅ <ø> (∅)
spanner-import-export ∅ <ø> (∅)
spanner-live-forward-migration ∅ <ø> (∅)
spanner-live-reverse-replication ∅ <ø> (∅)
spanner-bulk-migration ∅ <ø> (∅)
gcs-spanner-dv ∅ <ø> (∅)
see 954 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@derrickaw

derrickaw commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 15, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Apache Beam transform, ReadFromDeltaLake, to read data from Delta Lake tables, along with its corresponding unit tests and the deltalake dependency. The review feedback highlights critical improvements: replacing load_as_version with load_as_datetime when handling timestamp strings, and refactoring the table reading logic to use to_pyarrow_dataset().to_batches() to prevent Out Of Memory (OOM) errors on large datasets. Additionally, the unit tests should be updated to reflect these batching and datetime methods, and the storage configuration should avoid unsupported Hadoop keys like fs.gs.impl.

Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py Outdated
Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
@derrickaw

derrickaw commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 15, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Apache Beam transform, ReadFromDeltaLake, to read data from Delta Lake tables, along with its registration in the provider listing and the addition of the deltalake dependency. The review feedback highlights two critical runtime issues in the implementation: first, pyarrow.RecordBatch does not support to_pylist(), which will raise an AttributeError (using to_pydict() is suggested instead); second, loading a table by timestamp requires load_with_datetime rather than load_as_version. The reviewer also provided corresponding suggestions to update the unit tests to align with these fixes.

Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py Outdated
Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
@derrickaw
derrickaw marked this pull request as ready for review July 15, 2026 16:50
@gemini-code-assist

gemini-code-assist Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds support for reading data from Delta Lake tables into Apache Beam pipelines. It includes the necessary transform logic, updates the project dependencies to include the deltalake library, and provides unit tests to ensure the new functionality works as expected.

Highlights

  • New Delta Lake Transform: Introduced a new ReadFromDeltaLake PTransform and corresponding ReadDeltaLakeDoFn to enable reading data from Delta Lake tables within Apache Beam pipelines.
  • Dependency Management: Added deltalake as a new dependency in pyproject.toml and registered the new transform in the provider_listing_template.yaml.
  • Testing: Added comprehensive unit tests for both the ReadDeltaLakeDoFn and the ReadFromDeltaLake transform using mocking to verify functionality.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 15, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Apache Beam transform, ReadFromDeltaLake, along with its corresponding DoFn and unit tests, to enable reading data from Delta Lake tables. The review feedback highlights a critical runtime bug where pyarrow.RecordBatch.to_pylist() is called instead of to_pydict(), which was masked by mock objects in the tests. Additionally, the feedback suggests handling conflicting version and timestamp parameters by raising a ValueError and optimizing the DeltaTable initialization, alongside updating the unit tests to reflect these changes.

Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py Outdated
@derrickaw
derrickaw requested a review from chamikaramj July 15, 2026 16:53
@derrickaw derrickaw changed the title add readFromDeltaLake transform draft add readFromDeltaLake transform Jul 15, 2026
chamikaramj
chamikaramj reviewed Jul 15, 2026
View reviewed changes
Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py Outdated
@derrickaw
derrickaw requested a review from chamikaramj July 15, 2026 19:25
chamikaramj
chamikaramj reviewed Jul 17, 2026
View reviewed changes

@chamikaramj chamikaramj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

Comment thread python/src/test/python/job-builder-util-transforms/read_from_delta_lake_test.py
Comment thread python/src/main/python/job-builder-util-transforms/read_from_delta_lake.py
chamikaramj
chamikaramj approved these changes Jul 17, 2026
View reviewed changes

@chamikaramj chamikaramj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM.

@derrickaw
derrickaw merged commit ad677c0 into GoogleCloudPlatform:main Jul 17, 2026
20 checks passed
@derrickaw
derrickaw deleted the 20260714_deltaLakeAdd branch July 17, 2026 17:09
@derrickaw derrickaw mentioned this pull request Jul 17, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chamikaramj chamikaramj chamikaramj approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

addition New feature or request size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@derrickaw @chamikaramj